Supervised Neighborhoods for Distributed Nonparametric Regression

Techniques for nonparametric regression based on fitting small-scale local models at prediction time have long been studied in statistics and pattern recognition, but have received less attention in modern large-scale machine learning applications. In practice, such methods are generally applied to lowdimensional problems, but may falter with high-dimensional predictors if they use a Euclidean distance-based kernel. We propose a new method, Silo, for fitting predictiontime local models that uses supervised neighborhoods that adapt to the local shape of the regression surface. To learn such neighborhoods, we use a weight function between points derived from random forests. We prove the consistency of Silo, and demonstrate through simulations and real data that our method works well in both the serial and distributed settings. In the latter case, Silo learns the weighting function in a divide-andconquer manner, entirely avoiding communication at training time.